Method and system to develop a process improvement methodology

ABSTRACT

A method to create an instance of a defect-based production and testing process analysis machine (DPTPAM) provides continual process improvement based on foundational questions and classified defect data. The method includes the following steps: obtaining domain specific questions; developing a domain specific classification scheme that supports the answering of the foundational and domain specific questions; determining a method of using the domain specific classification scheme to answer both the foundational and domain specific questions; and creating a domain specific DPTPAM instance embodying the domain specific classification scheme and the method of answering the foundational and domain specific questions. The method can be implemented with a machine and a computer readable medium comprising logic for performing the method.

CROSS-REFERENCE TO RELATED APPLICATIONS

Not Applicable.

STATEMENT REGARDING FEDERALLY SPONSORED-RESEARCH OR DEVELOPMENT

Not Applicable.

INCORPORATION BY REFERENCE OF MATERIAL SUBMITTED ON A COMPACT DISC

Not Applicable.

FIELD OF THE INVENTION

The invention disclosed broadly relates to the field of informationprocessing systems, and more particularly relates to the field ofcustomizing a solution for process improvement based on defect analysis.

BACKGROUND OF THE INVENTION

Many defect-based process optimization techniques today are just binningschemes and do not link development and service. Patent applicationdescribes a method of providing production and test process analysisthat uses defect data as an input, but this patent application does notdescribe how such methods are created for a given customer.

One can attempt to reverse engineer the Orthogonal Defect Classification(ODC) scheme from existing instances, but the resulting methodologieswill not contain the right attributes and values to link the processeswith the field defects. Further, no guide exists explaining how thisreverse engineering should be done.

Thus, there remains a need for a method to develop a defect-basedprocess improvement methodology.

SUMMARY OF THE INVENTION

A method to create an instance of a defect-based production and testingprocess analysis machine (DPTPAM) provides continual process improvementbased on foundational questions and classified defect data. The methodincludes the following steps: obtaining domain specific questions;developing a domain specific classification scheme that supports theanswering of the foundational and domain specific questions; determininga method of using the domain specific classification scheme to answerboth the foundational and domain specific questions; and creating adomain specific DPTPAM instance embodying the domain specificclassification scheme and the method of answering the foundational anddomain specific questions. The method can be implemented with a machineand a computer readable medium comprising logic for performing themethod.

The foregoing and other objects, aspects, features, and advantages ofthe invention will become more apparent from the following descriptionand from the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an illustrative block diagram of the technology that wouldexecute as a result of execution of the creation method.

FIG. 2 is an illustrative diagram of the creation method execution.

FIG. 3 is an illustrative block diagram of the creation server accordingto one embodiment of the invention.

FIG. 4 is an illustrative logic diagram of the flow of the creationmethod.

FIG. 5 is an illustrative logic diagram for determining the submitterattributes

FIG. 6 is an illustrative logic diagram for determining the impactattributes

FIG. 7 is an illustrative logic diagram for determining the responderattributes

FIG. 8 is an illustrative logic diagram for determining the additionalattributes

FIG. 9 is an illustrative logic diagram for determining the scope of theanalysis

FIG. 10 is an illustrative logic diagram for determining the analysis

FIG. 11 is an illustrative block diagram of a creation method.

DETAILED DESCRIPTION

The following definitions will be used henceforth:

-   1) “Establishment of relationships and rules”—In deter-mining    analysis, we need to identify the attributes from the domain    specific classification scheme that will be used. For example, if we    want to evaluate test effectiveness, we might look at the following    classified data for defects: phase found vs. type of test, type of    test vs. conditions revealing defects, type of test vs. corrective    actions. One way we might look at this data is through bar charts.    The first chart would have phase found on the x axis and type of    test on the y-axis. We then need to identify what high and low    values mean—is a high value good or bad and what does it indicates    for process improvement.-   2) “In house” refers to finding defects by the development team at    their development location, before the product is released to the    customer. Customer or field defects refer to finding defects by the    customer.-   3) “Trending information” refers to looking for trends in the data    over time. We usually use open or closed date of the defects to look    at this kind of information.-   4) “Domain specific classification scheme” refers to the set of    attributes, each with own set own set of applicable values, used by    the customer to classify the defects found in the products they    produce, and whose classified data is used by the analysis rules to    produce an analysis report.-   5) “Foundational analysis” refers to analysis we will try to do for    any industry or customer, and which the domain specific    classification scheme must support. This includes:    -   a. Evaluating test efficiency/effectiveness (i.e., how well it        manages to catch defects that would have otherwise escaped);    -   b. Evaluating product stability (i.e., how reliability,        dependability, consistency);    -   c. Evaluating testing progress (i.e., how well each phase        catches its assigned the types of errors, e.g., parts testing        catching all part defects);    -   d. Determining phase (point) of defect injection (i.e.,        determining which production step produced which defects).    -   e. Evaluating strengths & weaknesses of process and service;-    (The first three are about removing defects; fourth is about    preventing defects; fifth is about both.)-   6) “Defect injection” refers to those phases that create defects in    the product. This can be done during the requirements phase if we    don't quite understand the customer's expectations, during the    design phase, or during actual build of the product—through coding    or assembly.-   7) “Testing” refers to attempts to remove defects from the product.    Many industries have multiple phases where they focus on defect    removal.

We now discuss an embodiment of the invention wherein a method providesa way to customize a solution for process improvement based on defectanalysis. This method creates a methodology for an industry whose goalis continual process improvement. The methodology includes creation of adomain specific classification scheme as well as a technique for doinganalysis of classified data. The defect data may include in processhardware or software defects, as well as field, service and warrantyclaims.

The embodiment described with reference to FIGS. 1-11 describes how thecurrent embodiment is used to provide a machine 3000 that producesinstances of a Defect-Based Production and Testing Process AnalysisMachine, DPTPAM 1000. First, a DPTPAM will be described, following whichthe creation method and associated machine 3000 will be described.

FIG. 1 is an illustrative flow control diagram of an instance of thetechnology 1000 that is produced as a result of an execution of a methodaccording to an embodiment of the invention, to create a Defect-BasedProduction and Testing Process Analysis Machine (DPTPAM). Given such aDPTPAM instance 1000, a customer is able to improve their developmentand testing process in the following manner. In step 1010 defects areobtained resulting from testing or from users of the product. Then, instep 1020, the defects are classified according to the domain specificclassification scheme created by the current embodiment 3000 (see FIG.3). This could involve the customer's responding to a series of promptsfor each defect, the prompts having the customer specify a value foreach of the scheme's attributes. The defects are then analyzed 1030according to the analysis determined by the creation machine 3000. Thiscould involve the DPTPAM 1000 creating charts for one or more of thescheme attributes. It might also involve determining if the classifieddefect contains patterns that match the analysis criteria, any matchesbeing noted. Finally, an analysis report is generated in step 1040 thataddresses the salient points of process improvement. This report caninclude the charts generated in step 1030, as well as indications of thepattern matches found. For example, if the associated pattern is found,the generated report would point out that the customer's product is notyet ready for release. This type of machine is described inYOR9-2005-0481.

Thus, a DPTPAM includes of the attributes and values that make up thedomain specific classification scheme, as well as the associatedanalysis necessary to determine how to improve the processes. Once theDPTPAM instance has been created, the user will use its classificationscheme to classify their attributes. Only one value for each of theattributes will be selected for classification. They will then use theresultant analysis to evaluate their development and testing process andidentify actions for improvement.

Skilled artisans will appreciate that although a DPTPAM could beembodied as an actual machine, e.g., a software application, the currentinvention also covers cases where the DPTPAM is embodied as a set ofinstructions to be carried out manually. These instructions would guidea given user through these steps 1010-1040.

FIG. 2 is an illustrative diagram an execution current invention. Asshown, first the creation method is executed 2010. This results in thecreation of a new DPTPAM instance 2020. Thus, the current invention is a(virtual) machine that is able to create a (virtual)machine—specifically a DPTPAM.

FIG. 3 depicts a block diagram of the computing system 3000 used in theexample embodiment that provides the current invention. This server 3000may comprise any computing node that is able to load and executeprogrammatic code, including, but not limited to: products sold by IBMsuch as ThinkPad® or PowerPC®, running the operating system and serverapplication suite sold by Microsoft, e.g., Windows® NT, or a Linuxoperating system. According to the present invention, the server logic3040 is preferably embodied as computer executable code that is loadedfrom a remote source (e.g., from a network file system), local permanentoptical (CD-ROM), magnetic storage (such as disk), or Storage 3020 intomemory 3030 for execution by CPU 3010. As will be discussed in greaterdetail below, the memory 3030 preferably includes computer readableinstructions, data structures, program modules and applicationinterfaces forming the following components: a Submitter Attribute andValue Handler 3050; an Impact Attribute and Value Handler 3060, aResponder Attribute and Value Handler 3070, an Additional Attribute andValue Handler 3080, a Scope Handler 3090, an Analysis Handler 3100,DPTPAM Creation Handler 3110, and a DPTPAM Creation Database 3120.

One of the outputs of a DPTPAM Creation Machine 3000 is a domainspecific classification scheme, a set of attributes, each with anassociated set of possible values, these attributes pertaining todefects of the product produced by the given customer. Under each ofthese attributes, appropriate values are identified. For each of theattributes and values identified by the DPTPAM Creation Machine 3000,there are certain rules that must be followed to optimize the knowledgeobtained from the attributes and values. First and foremost, theattributes and values representing the data collected must contribute toour knowledge about process improvement. If it does not, there is nosense in collecting it. In addition, for attributes, the data obtainedmust be orthogonal—non-redundant, non-overlapping, and mutuallyexclusive. That means any new attribute identified must first providenew data that has not been identified by the DPTPAM Creation Machine3000. We don't want to include data that is already captured elsewherein the scheme. In order to identify values under each of theseattributes, there are also rules that must be followed. The valuesshould be orthogonal also, non-overlapping and mutually exclusive. Inaddition the set of values for any attribute should be complete—that is,they should cover the entire range of data that could be collected forthat attribute. In some cases, the values cannot be orthogonal. In thiscase, it would then be important to assign a severity or priority foreach of the values, indicating which values should be selected first,should there be 2 equally correct possibilities.

The Submitter Attribute and Value Handler 3050, which is will bedescribed further with reference to FIG. 5, is responsible fordetermining the attributes and values related to the creation ofdefects. Typically, in an instance created by a DPTPAM Creation Machine3000, this information would be captured when a defect is opened by adesign engineer or tester and attempts to obtain information regardinghow the defect was found. This could include identifying attributes suchas “test phase that found the defect”, “type of test activity thatuncovered the defect”, or “environment or condition that revealed thedefect”, etc. . . . Under each of these attributes, appropriate valueswould then be identified, following the rules stated above. Thesubmitter attributes are important in analysis for evaluating testeffectiveness. This can be done whether defects are found before orafter release of a product. When using defects found before release, thesubmitter attributes are also used to measure progress. Therefore, theDPTPAM Creation Machine 3000 must identify attributes and values thatsupport this analysis, which we call the foundational analysis.

The Impact Attribute and Value Handler 3060, which is will be describedfurther with reference to FIG. 6, is responsible for determining theeffect or impact of the defect. If the defect was found before releaseof the product, the impact pertains to the possible impact of the defectif it had been found after release of the product. If the defect wasfound after release of the product, then the impact refers to the actualeffect of the defect. Impact may refer to the impact on the customer,the impact on the product, or the impact on a non-product such asproperty. For example, if the brakes fail on a car, their may be severalattributes or categories of impact. The failure may result in injury toa customer, injury to the product itself, and possibly injury toproperty if the car caused damage to a building. In this case, we wouldthen identify 3 different attributes for impact: customer, product, andnon-product. We would then have to define all the possible values foreach of those attributes following the rules stated above. For impact toa customer the DPTPAM Creation Machine 3000 might identify values of“death”, injury”, and “inconvenience.” Note that in this case, thevalues might not be orthogonal. A customer could experience injury anddeath. So they are not mutually exclusive. In this case, it would beimportant to assign a severity or priority to each value, then. Forexample, “death” would have the highest severity, followed by injury.Inconvenience would be assigned the lowest severity. During analysis,the impact attribute can be used to measure test effectiveness andproduct stability, as well as analysis specific to the domain.Therefore, when identifying attributes and values, these pieces of thefoundational analysis must be supported.

The Responder Attribute and Value Handler 3070, which is will bedescribed in detail with reference to FIG. 7, is responsible fordetermining the attributes and values that would be known when thedefect is fixed or resolved. During analysis this information is used todetermine where defects are being injected into the requirements,design, or building of the product. Analysis is also used to evaluateproduct stability and determine strengths and weaknesses of the process.Identified attributes may include “target”—the area that was fixed inresponse to the defect, scope of fix—what was the nature and scope ofthe fix, corrective action—what action was taken to fix the defect. Forscope of fix, it is important to obtain values that indicate not onlywhat the nature of the fix was, but also, the scope of the fix. Forexample, was a crack in the door handle simply repaired, or was thewhole transmission replaced? This is important information fordetermining where the defects are being injected, a part of theanalysis.

The Additional Attribute and Value Handler 3080, which is will bedescribed in detail with reference to FIG. 8, is responsible fordetermining any additional attributes and values specific to the domain(e.g., auto industry, healthcare, etc. . . ) that will contribute toprocess improvement of the said industry. These may serve to strengthenthe foundational analysis or they may identify a completely new area ofanalysis related to process improvement. For example, in the autoindustry, it was determined that we needed to capture an additionalattribute called “number of units affected,” in order to correctlyanalyze “place of defect injection.” Other DPTPAM instances 1000, onlyused “scope of defect fix” and “corrective action” to determine “placeof defect injection.”

The Scope Handler 3090, which is will be described in detail withreference to FIG. 9, determines the scope and type of attributes andvalues that can be created as well as the analysis that can be done.This will be dependent on the industry for which the DPTPAM CreationMachine 3000 is creating an instance. It will also be dependent on thetype of data that is available and the data that the customer is willingto collect in addition to any data that they may already collect. Thisdata could include: information on the manufacturing and testingprocess, defect data, Information on how the defects are fixed, andproduct information, such as cost, age of product.

The Scope Handler 3090 also determines which pieces of the foundationalanalysis can be performed. For example, if the customer is willing tocollect in house defect data pertaining to the test phase found, thetype of testing that uncovered the defect, and the conditions thatrevealed the defect, then evaluating test effectiveness and measuringprogress can be performed.

Jointly, the Submitter Attribute and Value Handler 3050, the ImpactAttribute and Value Handler 3060, the Responder Attribute and ValueHandler 3070, the Additional Attribute and Value Handler 3080, and theScope Handler 3090 create a given DPTPAM instance's domain specificclassification scheme.

The Analysis Handler 3100, which is will be described in detail withreference to FIG. 10, is responsible for creating the mapping ofattributes and values of the domain specific classification scheme toanalysis. It ensures that analysis rules are followed as well as sortingvalues into appropriate order to enable analysis.

The DPTPAM Creation Handler 3110 is responsible for the creation of theassociated DPTPAM instance 1000, i.e., one that used the domain specificclassification scheme built by the Submitter Attribute and Value Handler3050, the Impact Attribute and Value Handler 3060, the ResponderAttribute and Value Handler 3070, the Additional Attribute and ValueHandler 3080, and that uses the analysis determine by the Scope Handler3090 and Analysis Handler 3100. This creation includes: creation of code(or instructions) that guide a given customer through the classificationof their defects using the attributes and value scheme determined—i.e.,the DPTPAM instance's step 1020, creation of code (or instructions) thatanalyze the classified defect data using the of the rules that link theclassified defects to the analysis questions and answers, i.e., theDPTPAM instance's 1000 in step 1030, and creation of code (orinstructions) that produces an analysis report form, i.e., DPTPAMinstance's step 1040.

The DPTPAM Creation Database 3120 is responsible for storing theattributes, values, and queries that are created for the classificationscheme and analysis. The DPTPAM Creation Database 3120 can beimplemented using database tools such as the DB/2 product sold by IBM,and like database platforms.

FIG. 4 depicts the control flow of the DPTPAM Creation Machine's 3000logic 3040. As shown, first the Submitter Attribute and Value Handler3050 is executed to determine the attributes and values related to thecreation of defects. Next, the Impact Attribute and Value Handler 3060is executed which determines the effect or impact of given defects.Next, the Responder Attribute and Value Handler 3070 is executed, whichdetermines the attributes and values that would be known when a givendefect is fixed or resolved. Following this, Additional Attribute andValue Handler 3080 is executed, which determines any additionalattributes and values specific to a given the domain (e.g., autoindustry, healthcare, etc. . . ) that will contribute to processimprovement of the given domain. Next, the Scope Handler 3090 isexecuted, which determines the scope and type of attributes and valuesthat can be created as well as the analysis that can be done. Followingthis, the Analysis Handler 3100 is executed, which creates the mappingbetween the attributes and values and the analysis. Finally, the DPTPAMCreation Handler 3110 is executed, which created a DPTPAM instance 1000associated with the gathered attributes, values and relationships.

FIG. 5 depicts a logic diagram for determining submitter attributes3050. Submitter attributes are those that are filled in when a defect isfirst opened. The submitter attributes are determined through answers tospecific questions. The first question 6010 is “Are there multipletesting phases that are used to uncover defects?”Testing phases refer tocalendar or scheduled phases in a development process that uncoverdefects. If there are, then the names of these values will make up thevalues for Test Phase Found 6020. The next question 6030 to ask is “Arethere different types of testing that uncover defects?” If so, then theanswer to this question will make up the values for types of testing6040. Type of testing might include values like feature testing andsystem testing. Once you have determined the types of testing, then youmust identify the conditions revealing defects 6050. The values forconditions revealing defects reflect specific types of test cases orspecific types of environmental conditions that expose defects. Forexample, conditions could be environmental testing, or stress testing.They break the types of tests into more granular levels. Theseconditions should still adhere to the rules set up previously such thatthe conditions are a complete set of orthogonal values that help teamsdetermine ways to improve their testing process. Once the conditionshave been determined they must be assigned a category of test type 6060so for any specific type of test, there is a limited set of choices forcondition revealing defects. Once the conditions have been mapped, theflow ends 6070.

FIG. 6 depicts a logic diagram for determining impact attributes 3060.In this diagram, questions are asked to determine the impact of thedefects on customers that encounter them, on the product itself, and onany other entity, such as property. The first question 7010 asked is“Are there multiple ways the customer is impacted by defects?” If so,then the values will determine the customer impact 7020. Possible impactwould include performance, capability, reliability, security etc. . . .Next the question 7030 that must be answered is “Are there multiple waysthe product can be negatively affected by defects?” If the answer tothis question is yes, then the values 7040 for product impact can bedetermined. Values for product impact may include Fire, Theft, ProductInoperable, Function Inoperable, Safety Impairment, etc. . . . In thiscase, a priority scheme is used to sort the values with the highestpriority item. When the instance 1000 of the Creation Scheme is created,the user will classify this attribute starting at the highest priorityitem. For example, if the defect resulted in fire of the product, then“fire” will be selected for the value of Product Impact, because it isat the top of the list of values and has the highest severity orpriority. Finally, the question is asked “Are there non-product orpersonal impacts resulting from defects?” 7050. If yes, these valueswill determine the values for personal impact 7060, such as Death,Personal Injury, Environmental Damage, etc. . . . Once these values aredetermined, the program ends 7070.

FIG. 7 is a logic flow diagram for determining responder attributes3070. Responder attributes are those that are classified when the defectis fixed or closed. The first question 8010 that needs to be answered is“Are there multiple types of artifacts that get fixed when a defect iscorrected?” If yes, then these different types became the values 8020for “Target”. Here we are looking for high level areas that need to befixed in response to a defect being resolved. For example, when a defectis found in an automobile, you could fix the product itself (the car),or you could fix documentation related to the product, or you could fixthe transportation process, if many defects are introduced during thatprocess. So, in this case, the target values would be the product,documentation, and transportation process. The next set of questions isused to determine the scope or nature of the fix. Each target value willhave its own set of values for scope of fix. First, “Are there multipletypes of fixes that are applied?” 8030. If so, then we know there willbe multiple values for scope or nature of fix. The next 2 questions areimportant in identifying values for scope of fix that will allow us toanalysis pertaining to where in the development and testing process thedefects are being injected. “Are there distinct building phases?” 8040.These could include a requirements phase, design phase, building phase.These do not include test phases but are phase that function to build aproduct. These values will be the ones to be considered for where thedefects are being injected. The next question 8050 is “Are theredifferences in defects being injected in these phases? That means, canwe distinguish defects injected during the requirements phase vs.defects injected during the design phase vs. defects injected during thecoding, or prototype building phase? If yes, we will identify metricsthat allow us to determine which phase the defect was injected. Oncethese questions have been answered, we determine the values for scope offix for each target area. This is one of the more difficult areas as wemust keep in mind the answers to all the preceding questions. Inaddition to the rules mentioned for all scheme values, we also needvalues that a) capture complexity of fix from simple to more complex.For example, in a car, you may fix a simple part, a subsystem or asystem. b) Along with other metrics, can indicate which phases thedefects were injected. Finally, values for corrective action must alsobe determined 8070. That means for any defect scope of fix, what are allthe possible values that could be used for correcting the problem. Inthe case of the automotive industry, values included “Replace”,“Adjust”, “Install-New”, and “Reassemble”. These values will also beused in analysis to indicate stability of the product. Once these valueshave been determined, the responder attributes are complete 8080.

FIG. 8 is a logic diagram 3080 for determining any additional attributesfor either the submitter portion or the responder portion that areneeded to provide analysis. “Is there any additional domain specificdata that can be collected to support or enhance analysis and indicateprocess improvement?” 8070. If yes, then that data will be used todetermine attributes and values for the additional data 8080. Forexample, in the automotive industry, two additional attributesidentified were 1) Service Context and 2) # of Units Affected. ServiceContext includes values of “recall”, “routine maintenance”, and“unplanned maintenance”. These values will be important to see how manydefects were related to recall, routine or unplanned maintenance andwill indicate areas to focus on for improvement. For # of unitsaffected, it was determined that this number is needed to determinewhether defects were injected during the requirements, design, orprototyping phase. Once additional attributes have been identified andvalues created, then this task is complete 8090.

FIG. 9 is a logic diagram for determining the scope of analysis 3090.This will indicate the type of defect analysis can be performed based onthe limitations of the data so will indicate the focus when developinganalysis rules and relationships for the data. The first question asked9010 is whether defects are from software, hardware, or firmware. Ifdefects are not from any of these areas, we need go no further 9080because this invention creates a scheme for complex systems containinghardware or firmware and optionally software defects. If it isdetermined that this invention can be employed, then the next questionto answer is “Can defect data be collected pertaining to how the defectwas found?” 9020. This could include data from in house development ortest team or from the customer. But if there are details available abouthow and when the defect was first found, then metrics can be identifiedthat relate to phase found, types of testing, etc. . . 9040. Once thesemetrics are identified, then analysis can be formed to address testeffectiveness and any domain specific analysis related to how the defectwas found. In addition, if defect data is available from in housetesting phases, analysis rules and relationships can be establishedpertaining to entrance and exit criteria for testing phases andactivities and measuring progress throughout testing phases andactivities.

The next question is “Can defect data be collected pertaining to how thedefect was fixed?” 9030. If the answer is yes, then we determine theattributes and values for target, scope of fix, corrective action, andany other domain specific attributes needed 9050. Once we have theseattributes, we can establish relationships and rules for analysispertaining to product stability, phase of defect injection, etc. . . .9070. After these questions are answered, the metrics and analysis forresponder are complete 9080.

FIG. 10 is a logic diagram for determining analysis 3100. The attributesand values mentioned below are those from the domain specificclassification scheme. The first question asked 10010 is “Are attributevalues complete, non-overlapping, non-redundant, and orthogonal, with anindication of a level of complexity?” If the answer is yes, a level ofcomplexity is assigned 10020. Usually there are 4 levels with one beingthe least complex. The complexity is important in analysis as it helpsto measure progress during development and pinpoints where improvementsshould be made. In general, you want to see a trend from executing theless complex to the more complex. For “revealing conditions” attribute,for example, you want to see that defects have been executed though lesscomplex conditions first, and then c the more complex. One weakness thatis often present is that development teams find defects through thesimplex “revealing conditions” but never move on to the more complex,allowing the more complex defects to escape to the field and be found bycustomers. The next question 10030 is to ask “Do attribute values havean inherent priority? Usually, if they are not orthogonal and they donot have an inherent value of complexity, then they do need to have apriority 10040 assigned and the values need to be sorted by priority.For example, the attribute “Personal Impact” does not have a level ofcomplexity and is not orthogonal. Instead, the values are sorted bypriority, the value with the highest priority being at the top. Thevalue “Death” comes first, followed by “Personal Injury”, etc. . . .Once these questions are answered, then it is determined which of thefoundational analysis concepts can be supported with the data. The nextquestion 10050 is “Is Test Effectiveness relevant?” If data isavailable, on the how the defect was found such as condition revealingdefect, type of test, and test phase found for defects found in house,then we can evaluate test effectiveness during analysis. That means wehave to determine which attributes 10060 will be used to evaluate testeffectiveness. We also need to determine what high and low values meanwhen looking at aggregate data and interpret any trends. For example, ifthere are very high levels of the lower complexity “Conditions RevealingDefects, and low levels of the high complexity conditions, we need todetermine if this indicates high test effectiveness or low. This willdetermine the interpretation of the results obtained for the analysis.Next, the question 10070 is asked “Is Phase of Defect Injectionrelevant?” If there are multiple building phases where defects can beinjected into the product and we defined metrics for scope of fix andcorrective action, as well as other domain specific attributes, then weneed to determine exactly which attributes and values will be used inthis piece of analysis. We also need to determine what the high and lowvalues mean as well as any trending information. The next question 10090is “Is product stability is relevant?” One important criterion fordeveloping this type of analysis is that we have dates of when defectswere found and fixed so that we can do trending analysis. We want to seeif the product is becoming more or less stable over time. Of course, weneed other attributes as well, like severity of defect, whether thecorrective action was simply to replace a part of install a new one.Therefore, if product stability is relevant, we will determine therelationships 10100 of the attributes and values and interpret thetrends as well as the high and low values. Next, we ask 10110 if testprogress is relevant. Here we also need data for trending—either testphase found, or open date, for example. If test progress can be done, wedetermine the relationships 10120 of attributes and values andinterpretation of high and low values as well as any trendinginformation. Then we can move on to asking 10130 “If customer usagerelevant?” In this case, we need to have defects reported by customers,rather than defects found by the development team in house. If we havethat information and attributes that will address customer usage, thenwe can 10140 determine the relationship of those attributes and valuesas well as determining the meaning of high and low values. Also forcustomer usage, we will need dates for trending data. The final questionasked 10150 is “Is there any other domain analysis that should beconsidered to address customer concerns?” If previous analysis has notaddressed all customer concerns, then this is a place where we stillhave an opportunity to do that. Here 10160 we will need to determinewhich of the additional attributes and values can be used to addresscustomer concerns and interpret the high and low values. Once this hasbeen done, the development and interpretation of the analysis iscomplete 100170.

FIG. 11 is a block diagram illustrating an overview of the Defect-BasedProduction and Testing Process Analysis Method (DPTPAM) Creation Method.First, the domain specific classification scheme is developed in step5010, by obtaining domain specific questions, e.g. through executions ofthe Submitter Attribute and Value Handler 3050, the Impact Attribute andValue Handler 3060, the Responder Attribute and Value Handler 3070, theAdditional Attribute and Value Handler 3080, and the Scope Handler 3090respectively. Next, step 5020 develops a domain specific classificationscheme that supports the answering of the foundational and domainspecific questions, e.g., via executing steps 10150 and 10160 of theAnalysis Handler 3100. Next, step 5030 determines a method of using thedomain specific classification scheme to answer both the foundationaland domain specific questions, e.g., by executing the Analysis Handler3100. Note that this analysis could include the determination therelationships of attributes, the interpretation of high and low values,as well as any available trending information. Finally, step 5040creates a domain specific DPTPAM instance, embodying the domain specificclassification scheme and the method of answering the foundational anddomain specific questions, for the customer, e.g., by executing theDPTPAM Creation Handler 3110.

A skilled artisan will appreciate that a given service organizationcould use the current invention to provide DPTPAM-related services for afirst user. These DPTPAM-related services comprise: the serviceorganization creating all or part of a DPTPAM instance for the firstuser; the service organization updating all or part of the first user'sDPTPAM instance; the service organization validating (e.g., checking thecorrectness and completeness) all or part of the first user's DPTPAMinstance; the service organization analyzing of all or part of the firstuser's DPTPAM instance; the service organization teaching the first userto create all or part of a DPTPAM instance for themselves; the serviceorganization teaching the first user to update all or part of a DPTPAMinstance for themselves; the service organization teaching the firstuser to validate all or part of a DPTPAM instance for themselves; andthe service organization teaching the first user to analyze all or partof a DPTPAM instance for themselves.

1. A method to create an instance of a defect-based production andtesting process analysis machine (DPTPAM), the method comprising stepsof: obtaining domain specific questions, foundational questions, andclassified defect data; developing a domain specific classificationscheme that supports answering foundational and the domain specificquestions; determining a method of using the domain specificclassification scheme to answer both the foundational and domainspecific questions for performing a foundational specific analysis,wherein the analysis comprises: evaluating testefficiency/effectiveness; evaluating product stability; evaluating atesting progress; determining a phase of defect injection; andevaluating strengths and weaknesses of process and service; and creatinga domain specific DPTPAM instance embodying the domain specificclassification scheme and the method of answering the foundational anddomain specific questions for providing continual process improvementbased on the foundational questions and the classified defect data. 2.The method of claim 1, further comprising suggesting defect attributesto a customer to enable a more comprehensive defect analysis.
 3. Themethod of claim 1, further comprising determining a minimal set ofdefect attributes, maximally-complete set of values for each of thedefects.
 4. The method of claim 1, further comprising identifying defectattributes with properties of complexity that relate to progress of thedevelopment process.
 5. The method of claim 1, further comprisingdeveloping domain-specific attribute-value mappings and derivations sothat one set of attribute values can be automatically compute d from oneor more other known sets of attribute values.
 6. The method of claim 5,wherein developing additional attributes to enable the computation ofadding an ownership during attribute, so that test type can be computedgiven a condition exposing defect.
 7. The method of claim 1 furthercomprising validating a DPTPAM instance by using customer data.
 8. Themethod of claim 1, further comprising developing one specific schemethat can be used for defects found throughout including two or moreproduct life cycles, covering test, development and serviceorganizations.
 9. The method of claim 1 wherein data is grouped intosubmitter and responder parts, indicating that some information iscaptured when the defect is first opened and other attributes are filledin when the defect is closed.
 10. The method of claim 1 furthercomprising categorizing specific questions into major diagnostics areas,tasks, or groupings, providing domain expertise to user.
 11. The methodof claim 1 further comprising identifying stakeholders in the productlifecycle, the data they have available, and the analysis in which wouldbe interested.
 12. The method of claim 1 further comprising providingdynamic and static analysis.
 13. A machine for creating an instance of adefect-based production and testing process analysis machine (DPTPAM)for providing continual process improvement based on foundationalquestions and classified defect data, the machine comprising: aprocessor; a defect-based production and testing process analysismachine (DPTPAM) creation handler for creating a domain specific DPTPAMinstance; and a memory, coupled to the processor, and storing logic for:obtaining domain specific questions; developing a domain specificclassification scheme that supports the answering of the foundation anddomain specific questions; determining a method of using the domainspecific classification scheme to answer both the foundational anddomain specific questions; and creating the domain specific DPTPAMinstance embodying the domain specific classification scheme and themethod of answering the foundational and domain specific questions. 14.A computer program embodied on a computer readable medium, the computerprogram comprising program code for a DPTPAM creation handler forcreating a domain specific DPTPAM instance, the program code for:obtaining domain specific questions, foundational questions, andclassified defect data; developing a domain specific classificationscheme that supports the answering of the foundational questions and thedomain specific questions; determining a method of using the domainspecific classification scheme to answer both the foundational and thedomain specific questions for performing a foundational specificanalysis, wherein the analysis comprises: evaluating testefficiency/effectiveness; evaluating product stability; evaluating atesting progress; determining a phase of defect injection; andevaluating strengths and weaknesses of process and service; and creatinga domain specific DPTPAM instance embodying the domain specificclassification scheme and the method of answering the foundational andthe domain specific questions for providing continual processimprovement based on the foundational questions and the classifieddefect data.
 15. The computer program of claim 14 further comprising atleast one of: a submitter attribute and value handler; an impactattribute and value handler; an additional attribute and value handler;a scope handler; an analysis handler; and a DPTPAM creation database.